The present invention relates to an apparatus for sensing hand and finger position, orientation, and movement to interpret that information as it represents sign-language and outputting text and/or speech. This invention is specifically purposed toward the translation of sign language into sentences in the form of text and/or speech in a practical and ergonomic fashion to allow for easy and rapid communication.
Sign-language is prevalent among many different communities, such as the Deaf community. This method of communication is fast and effective for members of this community to communicate amongst themselves. However, like with every language, those who do not know sign-language face a significant barrier to communicating with users of sign-language. This barrier can impact both personal and professional opportunities. Providing sign-language using communities with the ability to rapidly communicate with those who do not understand sign-language through verbal and text speech could allow individuals to overcome these obstacles. What is needed is a way to bridge the gap between those who know sign language and people who can talk and hear who do not know sign-language.
The potential benefit of a way to facilitate better communication between these two groups is increasingly apparent when the sheer number of American Sign Language (ASL) speakers in the United States is considered. ASL is the native language of 500,000 to two million speakers in the United States alone. Outside of native speakers, an estimated 15 million people can communicate in sign language in the United States.
This hand motion interpretation and communication apparatus helps those who use sign-language as their primary form of communication to be able to communicate with other people who can hear but do not know sign language.
This hand motion interpretation apparatus aids those who do not know sign-language to learn more rapidly by hearing what words they are signing via the devices transliteration process and voice synthesizer.
Motion recognition and capture technology is advancing to become an integral part of everyday life. Specifically gesture recognition technology can be found in both recreational and military devices. The XBOX KINECT and the NINTENDO WII have infiltrated many people's homes and extensively utilize gesture capture software and technology to allow users to wirelessly interact with their devices. The military utilizes accurate gesture recognition (ANTRHOTRONIX NUGLOVE) to allow combat medics to effectively remote control robots, reducing the risk posed to medics in the field. Gesture recognition is becoming a more integral part of everyday life.
Sign-language gestures can be described in terms of three components: hand shape, location, and movement of the hands. Hand shape is the static component of a sign. This component comprises what is known as a “posture.” Sign-languages contain many of these unique postures, these postures are used to spell names or uncommon words that are not defined to a great extent in the dictionary. Even with these unique postures, sign-languages contain several signs that are very similar to each other. Those signs with a closed first are particularly difficult to distinguish with an untrained eye.
Due to the difficulties in visually distinguishing certain signs it can be difficult for visual based systems to properly recognize and interpret the signs.
There have been attempts to create hand motion interpretation apparatuses in the past, but a problem faced by these inventions is that their technology was not ergonomic nor practical for everyday use. The barriers to communication faced by the sign-language using community are everyday problems, and the technology must be convenient for that use. Physical based hand gesture interpretation apparatuses must not be cumbersome, restrictive, or unnatural to allow the user to easily create recognizable signs. Prior inventions serve their primary purposes in research and development. Additionally, prior hand gesture interpretation technology has not yet integrated advancements in computer processing and wireless communication.
As with other languages, every speaker or group of speakers will have small differences in their speech or signing. These differences can be found among signers of different ages, experience, or geographic location. The exact way the sign is done will vary but the underlying sign remains the same. Therefore, any interpretation system intended to recognize signs has to be able to classify signs, regardless of variation based on individual styles, accurately. In prior devices in this area of technology, a trade-off has had to be made, the sign-language user has sacrificed freedom of sign choice due to the limitations of the device.
Prior approaches have focused on methodologies for accomplishing sign-language transliteration: the hand alphabet which is used to fingerspell words, and complete signs which are formed by dynamic hand movements.
The outcome of these two methodologies was the creation of two classifications of inventions: video-based and instrumented. The video-based approach seemed to have a distinct advantage: the signer had complete freedom to sign and move without physical instruments attached to them. A camera with a limited field of view would monitor the hand movements and shape which limited the area of operation of signer. The signer would need to stay in the field of view of the camera or array of cameras. An additional limitation of this technology is the required processing power to handle the large amount of data created by video-based instruments. The complexity of the computer equipment required would be much higher as would be the expense.
On the other hand, to capture the dynamic nature of hand motions, it is necessary to have multiple sensors closely attached to a signer's hands or arm to measure position of the hand and fingers. For an instrumented approach, this has often included bulky instrumentation attached to a person. The data gathered by the sensors in such a device is also complex, requiring a physical connection to a desktop computer or laptop to transfer positional information. This physical connection to a computer limits the signer's freedom in multiple ways. A user of the device could not move far from the computer, nor could they physically move their arm freely as it is attached to the computer. In some cases, these gloves were complemented by infra-red, ultrasonic or magnetic trackers to capture movement and hand location. The drawback of these types of trackers is that they force the signer to remain close to the radiant source and inside a controlled environment free of interference (magnetic or luminescent) or interruptions of line of sight.
A number of hand motion recognition and interpretation systems have been proposed. Examples of these prior devices are disclosed in U.S. Pat. No. 5,699,441 to Sagawa et al., U.S. Pat. No. 8,140,339 to Hernandez-Rebollar, U.S. Pat. No. 9,098,493 to Tardif.